Design, methods, and reporting of impact studies of cardiovascular clinical prediction rules are suboptimal: a systematic review.

OBJECTIVES: To evaluate design, methods, and reporting of impact studies of cardiovascular clinical prediction rules (CPRs). STUDY DESIGN AND SETTING: We conducted a systematic review. Impact studies of cardiovascular CPRs were identified by forward citation and electronic database searches. We categorized the design of impact studies as appropriate for randomized and nonrandomized experiments, excluding uncontrolled before-after study. For impact studies with appropriate study design, we assessed the quality of methods and reporting. We compared the quality of methods and reporting between impact and matched control studies. RESULTS: We found 110 impact studies of cardiovascular CPRs. Of these, 65 (59.1%) used inappropriate designs. Of 45 impact studies with appropriate design, 31 (68.9%) had substantial risk of bias. Mean number of reporting domains that impact studies with appropriate study design adhered to was 10.2 of 21 domains (95% confidence interval, 9.3 and 11.1). The quality of methods and reporting was not clearly different between impact and matched control studies. CONCLUSION: We found most impact studies either used inappropriate study design, had substantial risk of bias, or poorly complied with reporting guidelines. This appears to be a common feature of complex interventions. Users of CPRs should critically evaluate evidence showing the effectiveness of CPRs.

Factors associated with evidence-based decision-making among patients and providers.

Aim: For comparative effectiveness research to achieve its purpose, providers and patients must use research evidence to make medical decisions. Therefore, this study examined factors associated with evidence-based decision-making by patients and providers. Methods: Data were collected via cross-sectional online surveys of patients (n = 603) and providers (n = 628) between November 2011 and January 2012. Results: For both patients and providers, evidence-based medical decision-making is associated with perceptions, that is, some combination of self efficacy, attitudes and opinions. However, whereas knowledge is the most consistent factor associated with decision-making for providers, it is not associated at all for patients. Conclusion: Efforts to promote evidence-based medical decision-making among patients and providers should focus on skills training to improve self efficacy, and messages that highlight the benefits of patient engagement in medical decisions.

A Structured Preapproval and Postapproval Comparative Study Design Framework to Generate Valid and Transparent Real-World Evidence for Regulatory Decisions.

Real-world evidence provides important information about the effects of medicines in routine clinical practice. To engender trust that evidence generated for regulatory purposes is sufficiently valid, transparency in the reasoning that underlies study design decisions is critical. Building on existing guidance and frameworks, we developed the Structured Preapproval and Postapproval Comparative study design framework to generate valid and transparent real-world Evidence (SPACE) as a process for identifying design elements and minimal criteria for feasibility and validity concerns, and for documenting decisions. Starting with an articulated research question, we identify key components of the randomized controlled trial needed to maximize validity, and pragmatic choices are considered when required. A causal diagram is used to justify the variables identified for confounding control, and key decisions, assumptions, and evidence are captured in a structured way. In this way, SPACE may improve dialogue and build trust among healthcare providers, patients, regulators, and researchers.

Real-world data: towards achieving the achievable in cancer care.

The use of data from the real world to address clinical and policy-relevant questions that cannot be answered using data from clinical trials is garnering increased interest. Indeed, data from cancer registries and linked treatment records can provide unique insights into patients, treatments and outcomes in routine oncology practice. In this Review, we explore the quality of real-world data (RWD), provide a framework for the use of RWD and draw attention to the methodological pitfalls inherent to using RWD in studies of comparative effectiveness. Randomized controlled trials and RWD remain complementary forms of medical evidence; studies using RWD should not be used as substitutes for clinical trials. The comparison of outcomes between nonrandomized groups of patients who have received different treatments in routine practice remains problematic. Accordingly, comparative effectiveness studies need to be designed and interpreted very carefully. With due diligence, RWD can be used to identify and close gaps in health care, offering the potential for short-term improvement in health-care systems by enabling them to achieve the achievable.

Time-dependent biases in observational studies of comparative effectiveness research in rheumatology. A methodological review.

OBJECTIVE: To assess to what extent time-dependent biases (ie, immortal time bias (ITB) and time-lag bias (TLB)) occur in the latest rheumatology observational studies, describe their main mechanisms and increase the awareness on this topic. METHODS: We searched PubMed for observational studies on rheumatic diseases published in leading medical journals in the last 5 years. Only studies with a time-to-event analysis exploring the association of one or more interventional strategies with an outcome were included. Each study was labelled as free from bias, at risk of TLB, at risk of misclassified ITB if the period of immortal time was incorrectly attributed to an intervention group, or at risk of excluded ITB if the immortal time was discarded from the analysis. RESULTS: We included 78 papers. Most studies were performed in Europe or North America (46% each), were not industry funded (62%) and had a safety primary outcome (59%). In total, 13 (17%) studies were considered at risk of time-dependent biases. Among the studies at risk of ITB (n=8; 10%), in 5 (6%), waiting time to receive treatment was wrongly attributed to the treatment exposure group, which indicated misclassified ITB. Five (6%) studies were at risk of TLB: patients on conventional synthetic disease-modifying antirheumatic drugs (DMARD; first-line drugs) were compared with patients on biologic DMARDs (second or third-line drugs) without accounting for disease duration or prior medication use. CONCLUSIONS: One in six comparative effectiveness observational studies published in leading rheumatology journals is potentially flawed by time-dependent biases.

Hypothesis Testing in Superiority, Noninferiority, and Equivalence Clinical Trials: Implications in Physical Medicine and Rehabilitation.

In medical research, it is important to be able to examine whether there is a significant difference between two samples. With this, establishing an appropriate hypothesis is a critical, basic step for correct interpretation of results in inferential statistical data analysis. It is important to note that the aim of hypothesis testing is not to "accept" or "reject" the null hypothesis but to gauge the likelihood that the observed difference is genuine if the null hypothesis is true.Traditionally, the null hypothesis assumes that there is no statistically significant difference between the two groups. It has become more difficult to develop new treatments that are better than the standard of care. This review article summarizes and explains the methodology of the different types of clinical trials regarding the relevant basic statistical concepts and hypothesis testing.

Choosing important health outcomes for comparative effectiveness research: 4th annual update to a systematic review of core outcome sets for research.

BACKGROUND: The Core Outcome Measures in Effectiveness Trials (COMET) database is a publically available, searchable repository of published and ongoing core outcome set (COS) studies. An annual systematic review update is carried out to maintain the currency of database content. METHODS: The methods used in the fourth update of the systematic review followed the same approach used in the original review and previous updates. Studies were eligible for inclusion if they reported the development of a COS, regardless of any restrictions by age, health condition or setting. Searches were carried out in March 2018 to identify studies that had been published or indexed between January 2017 and the end of December 2017. RESULTS: Forty-eight new studies, describing the development of 56 COS, were included. There has been an increase in the number of studies clearly specifying the scope of the COS in terms of the population (n = 43, 90%) and intervention (n = 48, 100%) characteristics. Public participation has continued to rise with over half (n = 27, 56%) of studies in the current review including input from members of the public. The rate of inclusion of all stakeholder groups has increased, in particular participation from non-clinical research experts has risen from 32% (mean average in previous reviews) to 62% (n = 29). Input from participants located in Australasia (n = 17; 41%), Asia (n = 18; 44%), South America (n = 13; 32%) and Africa (n = 7; 17%) have all increased since the previous reviews. CONCLUSION: This update included a pronounced increase in the number of new COS identified compared to the previous three updates. There was an improvement in the reporting of the scope, stakeholder participants and methods used. Furthermore, there has been an increase in participation from Australasia, Asia, South America and Africa. These advancements are reflective of the efforts made in recent years to raise awareness about the need for COS development and uptake, as well as developments in COS methodology.

Maintaining Internal Validity in Community Partnered Participatory Research: Experience from the Community Partners in Care Study.

Objective: With internal validity being a central goal of designed experiments, we seek to elucidate how community partnered participatory research (CPPR) impacts the internal validity of public health comparative-effectiveness research. Methods: Community Partners in Care (CPIC), a study comparing a community-coalition intervention to direct technical assistance for disseminating depression care to vulnerable populations, is used to illustrate design choices developed with attention to core CPPR principles. The study-design process is reviewed retrospectively and evaluated based on the resulting covariate balance across intervention arms and on broader peer-review assessments. Contributions of the CPIC Council and the study's design committee are highlighted. Results: CPPR principles contributed to building consensus around the use of randomization, creating a sampling frame, specifying geographic boundaries delimiting the scope of the investigation, grouping similar programs into pairs or other small blocks of units, collaboratively choosing random-number-generator seeds to determine randomized intervention assignments, and addressing logistical constraints in field operations. Study protocols yielded samples that were well-balanced on background characteristics across intervention arms. CPIC has been recognized for scientific merit, has drawn attention from policymakers, and has fueled ongoing research collaborations. Conclusions: Creative and collaborative fulfillment of CPPR principles reinforced the internal validity of CPIC, strengthening the study's scientific rigor by engaging complementary areas of knowledge and expertise among members of the investigative team.

Validation protocols for blood pressure measuring devices in the 21st century.

Blood pressure (BP) is a vital sign and the essential measurement for the diagnosis of hypertension. Therefore, its accurate measurement is a key element for the evaluation of many medical conditions and for the reliable diagnosis and efficient treatment of hypertension. In the last 3 decades prestigious organizations, such as the US Association for the Advancement of Medical Instrumentation (AAMI), the British Hypertension Society, the European Society of Hypertension (ESH) Working Group on BP Monitoring, and the International Organization for Standardization (ISO), have developed protocols for clinical validation of BP measuring devices. All these initiatives aim to standardize validation procedures and establish minimum accuracy standards for BP monitors. Unfortunately, only a few of the BP measuring devices available on the market have been subjected to independent validation using one of these protocols. Recently, the AAMI, ESH, and ISO experts agreed to develop a single universally acceptable standard (AAMI/ESH/ISO), which will replace all previous protocols. This major international initiative has been undertaken to best serve the needs of patients with hypertension, a public interested in cardiovascular health, practicing physicians, scientific researchers, regulatory bodies, and manufacturers. There is an urgent need to influence regulatory authorities throughout the world to make it mandatory for all BP measuring devices to have undergone independent validation before approval for marketing. Efforts need to be intensified to improve the accuracy of BP measuring devices, further optimize the validation procedure, and ensure that objective and unbiased validation data become available.

Comparison of pregnancy rates for poor responders using IVF with mild ovarian stimulation versus conventional IVF: a guideline.

Mild-stimulation protocols with in vitro fertilization (IVF) generally aim to use less medication than conventional IVF. This guideline evaluates pregnancy and live-birth rates in patients expected to be poor responders using mild ovarian stimulation and natural-cycle protocols vs conventional IVF.

Evaluation of comparative effectiveness research: a practical tool.

Comparative effectiveness research (CER) guidelines have been developed to direct the field toward the most rigorous study methodologies. A challenge, however, is how to ensure the best evidence is generated, and how to translate methodologically complex or nuanced CER findings into usable medical evidence. To reach that goal, it is important that both researchers and end users of CER output become knowledgeable about the elements that impact the quality and interpretability of CER. This paper distilled guidance on CER into a practical tool to assist both researchers and nonexperts with the critical review and interpretation of CER, with a focus on issues particularly relevant to CER in oncology.

Comparative effectiveness research methodology using secondary data: A starting user's guide.

BACKGROUND: The use of secondary data, such as claims or administrative data, in comparative effectiveness research has grown tremendously in recent years. PURPOSE: We believe that the current review can help investigators relying on secondary data to (1) gain insight into both the methodologies and statistical methods, (2) better understand the necessity of a rigorous planning before initiating a comparative effectiveness investigation, and (3) optimize the quality of their investigations. MAIN FINDINGS: Specifically, we review concepts of adjusted analyses and confounders, methods of propensity score analyses, and instrumental variable analyses, risk prediction models (logistic and time-to-event), decision-curve analysis, as well as the interpretation of the P value and hypothesis testing. CONCLUSIONS: Overall, we hope that the current review article can help research investigators relying on secondary data to perform comparative effectiveness research better understand the necessity of a rigorous planning before study start, and gain better insight in the choice of statistical methods so as to optimize the quality of the research study.

Secondary data analysis: techniques for comparing interventions and their limitations.

PURPOSE OF REVIEW: Secondary data analysis has become increasingly common in health services research, specifically comparative effectiveness research. While a comprehensive study of the techniques and methods for secondary data analysis is a wide-ranging topic, we sought to perform a descriptive study of some key methodological issues related to secondary data analyses and to provide a basic summary of techniques to address them. RECENT FINDINGS: In this study, we first address common issues seen in analysis of secondary datasets, and limitations of datasets with respect to bias. We cover some strategies for handling missing or incomplete data and a basic summary of three statistical approaches that can be used to address the problem of bias. SUMMARY: While it is unrealistic for surgeon scientists to aspire to the depth of knowledge of professional statisticians or data scientists, it is important for researchers and clinicians reading to understand some of the common pitfalls and issues when using secondary data to investigate clinical questions. Ultimately, the choice of analytical technique and the particular data sets used should be dictated by the research question and hypothesis being tested. Transparency about data handling and statistical techniques are vital elements of secondary data analysis.

Grading evidence from test accuracy studies: what makes it challenging compared with the grading of effectiveness studies?

Guideline panels need to process a sizeable amount of information to issue a decision on whether to recommend a health technology or not. Grading of Recommendations Assessment, Development, and Evaluation (GRADE) is being frequently applied in guideline development to facilitate this task, typically for the synthesis of effectiveness research. Questions regarding the accuracy of medical tests are ubiquitous, and they temporally precede questions about therapy. However, literature summarising the experience of applying GRADE approach to accuracy evaluations is not as rich as one for effectiveness evidence. Type of study design (cross-sectional), two-dimensional nature of the performance measures (sensitivity and specificity), propensity towards a higher level of between-study heterogeneity, poor reporting of quality features and uncertainty about how best to assess for publication bias among other features make this task challenging. This article presents solutions adopted to addresses above challenges for judicious estimation of the strength of test accuracy evidence used to inform evidence syntheses for guideline development.

Assessment of Confounders in Comparative Effectiveness Studies From Secondary Databases.

Secondary clinical databases are an important and growing source of data for comparative effectiveness research (CER) studies. However, measurement of confounders, such as biomarker values or patient-reported health status, in secondary clinical databases may not align with the initiation of a new treatment. In many published CER analyses of registry data, investigators assessed confounders based on the first questionnaire in which the new exposure was recorded. However, it is known that adjustment for confounders measured after the start of exposure can lead to biased treatment effect estimates. In the present study, we conducted simulations to compare assessment strategies for a dynamic clinical confounder in a registry-based comparative effectiveness study of 2 therapies. As expected, we found that adjustment for the confounder value at the time of the first questionnaire after the start of exposure creates a biased estimate the total effect of exposure choice on outcome when the confounder mediates part of the effect. However, adjustment for the prior value can also be badly biased when measured long before exposure initiation. Thus, investigators should carefully consider the timing of confounder measurements relative to exposure initiation and the rate of change in the confounder in order to choose the most relevant measure for each patient.

Policies for Use of Real-World Data in Health Technology Assessment (HTA): A Comparative Study of Six HTA Agencies.

BACKGROUND: Randomized controlled trials provide robust data on the efficacy of interventions rather than on effectiveness. Health technology assessment (HTA) agencies worldwide are thus exploring whether real-world data (RWD) may provide alternative sources of data on effectiveness of interventions. Presently, an overview of HTA agencies' policies for RWD use in relative effectiveness assessments (REA) is lacking. OBJECTIVES: To review policies of six European HTA agencies on RWD use in REA of drugs. A literature review and stakeholder interviews were conducted to collect information on RWD policies for six agencies: the Dental and Pharmaceutical Benefits Agency (Sweden), the National Institute for Health and Care Excellence (United Kingdom), the Institute for Quality and Efficiency in Healthcare (Germany), the High Authority for Health (France), the Italian Medicines Agency (Italy), and the National Healthcare Institute (The Netherlands). The following contexts for RWD use in REA of drugs were reviewed: initial reimbursement discussions, pharmacoeconomic analyses, and conditional reimbursement schemes. We identified 13 policy documents and 9 academic publications, and conducted 6 interviews. RESULTS: Policies for RWD use in REA of drugs notably differed across contexts. Moreover, policies differed between HTA agencies. Such variations might discourage the use of RWD for HTA. CONCLUSIONS: To facilitate the use of RWD for HTA across Europe, more alignment of policies seems necessary. Recent articles and project proposals of the European network of HTA may provide a starting point to achieve this.